Papers with weight tying
Joey NMT: A Minimalist NMT Toolkit for Novices (D19-3)
Copied to clipboard
| Challenge: | a recent study shows that novices perform better than experts in a code quiz. |
| Approach: | They present a minimalist neural machine translation toolkit based on PyTorch . they evaluate the accessibility of the toolkit in a user study . |
| Outcome: | The proposed toolkit performs comparable to more complex toolkits on standard benchmarks. |
Revisiting Representation Degeneration Problem in Language Modeling (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Language modeling is a fundamental task in natural language processing, applications include machine translation, image captioning and speech recognition. |
| Approach: | They propose a cosine regularization method to solve the representation degeneration problem by analyzing the limitations of the proposed method and then propose an alternative regularization technique to tackle the problem. |
| Outcome: | The proposed method is effective in language modeling and image captioning. |
Reproducing and Regularizing the SCRN Model (C18-1)
Copied to clipboard
| Challenge: | Recurrent neural networks (RNNs) have demonstrated tremendous success in sequence modeling . naive dropout, variational dropout and weight tying are common techniques used to regularize the SCRN model . |
| Approach: | They propose a Structurally Constrained Recurrent Network (SCRN) model and regularize it using existing techniques. |
| Outcome: | The proposed model outperforms the LSTM model on non-English data while being much simpler. |
Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight Tying (2024.naacl-long)
Copied to clipboard
| Challenge: | a new paradigm for low-rank Adaptation (LoRA) uses weight tying and selective training to improve parameter efficiency. |
| Approach: | They propose a paradigm that uses weight tying and selective training to enhance parameter efficiency of Low-rank Adaptation. |
| Outcome: | The proposed paradigm achieves comparable performance to LoRA with reduced model complexity . the proposed paradigm can be used for a variety of tasks and languages . |
Weight Tying Biases Token Embeddings Towards the Output Space (2026.findings-acl)
Copied to clipboard
| Challenge: | Weight tying is a common practice in language model design, but its impact on learning embedding space remains unclear. |
| Approach: | They show that weight tying optimizes the embedding matrix for output prediction . they also show that tied embeddable matrices align more closely with output embedders . |
| Outcome: | The proposed weight tying approach harms performance at scale and has implications for training smaller LLMs. |